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EXECUTIVE  SUMMARY 


Cybersecurity  is  a  growing  concern  for  the  U.S.  Government,  indeed  the  U.S.  is  on  the  receiving  end  of 
an  estimated  100,000  cyber-attacks  each  day.  Cybersecurity  is  a  fast-growing  market  where  technologies 
are  constantly  evolving  to  counter  threats  to  information  and  operational  systems.  Across  the  U.S. 
Government  as  a  whole,  there  is  no  standardized  and  repeatable  methodology  for  evaluating  cybersecurity 
technologies.  In  this  report,  we  introduce  the  Department  of  Defense  (DoD)-centric  and  Independent 
Technology  Evaluation  Capability  (DITEC),  an  experimental  decision  support  service  within  the  U.S.  DoD 
which  aims  to  provide  a  standardized  framework  for  cybersecurity  technology  evaluations  in  support  of 
acquisition  decision  making.  In  addition  to  DITEC  as  a  proof-of-concept,  we  describe  a  family  of  services 
that  include:  DITEC-i-,  an  enterprise-level  tool,  and  the  Cyber-Supervisory  Control  and  Data  Acquisition 
(SCADA)  Evaluation  Capability  (C-SEC),  an  instantiation  of  DITEC  for  evaluating  SCADA  network 
cybersecurity  technologies. 
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1.  INTRODUCTION 


Cybersecurity  is  a  growing  concern  for  the  U.S.  Government,  indeed  the  U.S.  is  on  the  receiving  end  of 
an  estimated  100,000  cyber-attacks  each  day  [1].  Cybersecurity  technology  is  a  fast-growing  market  where 
technologies  are  constantly  evolving  to  counter  threats  to  information  and  operational  systems. 
Cybersecurity  investment  processes  for  private  and  public  sector  organizations  vary  considerably 
— government  agencies  are  often  constrained  by  long-term  budgetary  and  acquisition  procedures  while 
private  sector  organizations  can  make  purchasing  and  policy  decisions  with  considerably  more  ease  [2]. 
Across  the  U.S.  Government  as  a  whole,  there  is  no  standardized  and  repeatable  methodology  for 
evaluating  cybersecurity  technologies.  As  a  consequence,  the  cybersecurity  acquisition  process  inevitably 
leads  to  duplicated  efforts  on  the  part  of  technical  and  acquisition  personnel.  Moreover,  lessons  learned 
within  one  sector  of  the  government  are  not  easily  shared  with  others,  which  may  lead  to  multiple  agencies 
adopting  a  cybersecurity  technology  that  fails  to  meet  their  needs  [3].  In  certain  sectors  of  the  government, 
where  personnel  are  rotated  through  on  a  regular  basis,  cybersecurity  policies  and  products  may  be 
replaced  with  each  change  in  project  supervision.  A  practical  and  important  consequence  of  this  is  that 
cybersecurity  acquisition  decisions  will  be  made  by  security  non-experts.  If  an  expert  in  a  security  topic 
leaves  a  team,  their  institutional  knowledge  on  that  topic  may  be  lost.  Knowledge  that  was  common  sense 
in  previous  decision-making  efforts  is  not  obvious  to  new  teams.This  situation  introduces  unacceptable 
risks  such  as  the  following: 

•  The  decision  to  acquire  a  new  technology  or  product  may  involve  outdated  knowledge,  due  to  the 
quickly  evolving  nature  of  cybersecurity  technologies 

•  Chosen  products  or  technologies  may  not  be  the  optimal  choice  for  a  system’s  security  needs 

•  Network  administrators  must  learn  to  use  the  new  products  which  have  been  acquired. 

We  describe  the  Department  of  Defense  (DoD)-centric  and  Independent  Technology  Evaluation 
Capability  (DITEC),  a  U.S.  Department  of  the  Navy  research  project  which  aims  to  rectify  these  problems 
and  provide  decision  support  for  cybersecurity  acquisition  professionals,  and  its  family  of  software 
products.  DITEC  and  its  family  of  products  offer  a  standardized  and  repeatable  platform  for  performing 
and  preserving  technology  and  product  evaluations,  yet  incorporate  the  flexibility  to  score  products  for  use 
in  different  contexts.  The  rest  of  this  work  is  organized  as  follows:  Section  2  discuses  policies  and 
procedures  for  cybersecurity  acquisition;  Section  3  describes  DITEC  in  depth;  Section  4  describes 
DITEC-I-,  an  enterprise-ready  implementation  of  DITEC;  Section  5  describes  an  instantiation  of  DITEC-i- 
used  for  evaluating  cybersecurity  technologies  in  an  operational  technology  environment  as  well  as 
in-depth  descriptions  of  features  and  functionality;  Sections  6  and  7  cover  future  work  and  concluding 
remarks,  respectively. 
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2.  CYBERSECURITY  TECHNOLOGY  ACQUISITION: 

POLICY  AND  PROCEDURE 

Billions  of  dollars  are  invested  in  eyberseeurity  teehnology  eaeh  year  [4].  This  money  is  spent  on 
procurement,  maintenance,  retirement,  and  replacement  of  these  products.  The  acquisition  process  is  often 
inefficient,  resulting  in  increased  costs  for  a  delayed  product  that  may  not  adequately  meet  the  security 
needs  of  the  organization. 

2.1  THE  DECISION  TO  PROCURE 

In  large  organizations,  the  decision  to  procure  a  eyberseeurity  solution  is  often  driven  by  requirements 
rather  than  by  an  observed  specific  need.  Conversely,  once  a  real  need  for  a  solufion  appears  if  may  nof  be 
clearly  supporfed  by  a  defined  requiremenf.  Furfhermore,  requesfs  which  are  nof  clearly  supported  by 
requiremenfs  in  fhe  organizafion’s  chosen  framework  may  nof  be  supported  by  senior  managemenf  and  fhe 
burden  fo  prove  ifs  necessify  will  resf  wifh  fhe  chief  informalion  security  officer  (CISO)  or  ofher 
informafion  securify  managers. 

Many  organizations  use  fhe  NIST‘,  ISO-27000^,  or  COBIT  frameworks^  buf  for  smaller  organizafions, 
fhese  frameworks  can  be  overly  complex  and  may  nof  meef  fhe  organizafion’s  percepfion  of  risk.  As  such, 
fhe  need  for  a  cyber  solufion  may  be  even  more  ambiguously  supporfed  by  fhe  adopfed  requiremenfs. 

Once  a  cyber  solufion  is  identified  as  needed  by  informafion  securify  personnel  (even  if  funding  is 
available)  fhe  procuremenf  requesf  is  likely  fo  meef  resisfance  from  senior  managemenf  if  fhey  believe  fhe 
organizafion  does  nof  have  fhe  sfaffing  or  experience  necessary  fo  execufe  fhe  solufion.  A  lack  of 
experience  or  personnel  may  also  drive  a  deparfmenf  fo  nof  adopf  a  fechnology  if  fhey  feel  fhaf  ifs 
capabilifies  mighf  be  greater  fhan  fheir  own  capabilifies  fo  fully  undersfand  fhe  solufion,  even  if  if  is 
adequafe  fo  meef  fheir  needs.  A  lack  of  mafure  refurn  on  invesfmenf  (ROI)  models  for  eyberseeurity  leaves 
some  deparfmenfs  scrambling  fo  research  financial  dafa  if  fheir  organizafion  is  more  concerned  wifh 
proving  a  financial  benefil  for  adopting  a  cybersecurify  fechnology. 

2.2  TECHNOLOGY  SELECTION 

Oflen  organizafions  will  adopf  frameworks  (e.g.,  NIST,  ISO-27000,  or  COBIT)  fo  drive  cybersecurify 
decision  making.  These  frameworks  do  benefif  organizafions,  as  if  has  been  shown  fhaf  fhe  lack  of  an 
adopfed  a  framework  correlates  fo  underspending  on  cybersecurify.  However,  nof  all  cyber  acquisifion  can 
be  easily  defined  or  addressed  by  fhe  requiremenfs  of  fhaf  adopfed  framework  [2]. 

Governmenf  agencies,  which  are  heavily  compliance-driven,  do  well  in  identifying  fhe  need  for  cyber 
acquisifion  buf  nof  in  idenfifying  fhe  besf  fechnology  fo  meef  fhaf  need  [2] .  Despile  fhe  vasl  sums  of  money 
spenl  on  cybersecurify,  Ihere  is  no  slandardized  melhod  fo  assisl  cybersecurify  personnel  in  choosing  fhe 
mosf  appropriafe  producfs  fo  secure  fheir  syslems  [3].  The  focus  is  often  more  on  checking  boxes  fhan 
evaluating  cyber  risks. 

Once  a  fechnology  type  is  idenlified,  if  is  sfill  very  difficull  fo  determine  which  vendor  producl  will 
perform  mosf  efficienlly  wifhouf  performing  a  bulk  comparison  of  technologies  againsf  each  ofher.  Oflen, 
organizafions  will  slarl  fhe  comparison  via  Ihird-parlies  such  as  Garlner"'  or  Forresler^  [2],  however  nof  all 

*  https  ://www.nist.gov/cyberframework 
^http://standards.iso.org/ 

^http://www.isaca.org/ 

"'http://www.gartner.com/ 

^https://go. forrester.com/ 
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organizations  have  the  funding  or  resourees  to  proeure  and  set  up  each  technology  and  test  it  against 
real-world  threats  just  to  make  a  purchasing  decision.  Moreover,  not  all  organizations  that  have  a  need  for  a 
technology  are  qualified  to  determine  how  well  one  vendor’s  product  performs  against  others. 

2.3  VENDOR  SELECTION 

During  an  organization’s  decision-making  process,  vendors  will  often  write  project  proposals  for  their 
own  products — either  those  on  the  market  or  in  development.  This  often  results  in  the  decision  of  which 
cybersecurity  technology  to  purchase  being  inadvertently  made  by  the  vendors  themselves. 

The  contract,  which  is  detailed,  extremely  inflexible,  and  difficult  to  modify,  requires  the  products  listed 
in  the  vendor’s  proposal  with  little  concern  for  emerging  threats  or  those  products’  future  performance. 
Once  the  proposal  is  accepted,  the  organization  is  limited  to  that  vendor’s  product  offering,  even  as  the 
cyber  domain  changes.  The  capabilities  selected  are  limited  to  only  those  the  vendor  can  provide,  and  the 
vendor  is  required  to  produce  only  what  is  proposed.  This  forces  the  vendor  to  produce  only  contracted 
solutions  rather  than  incorporating  new  technology  to  address  new  cyber  threats.  While  contracts  are  being 
worked  out,  the  organization  is  effectively  limited  in  its  capability  to  handle  any  new  threats. 

Additionally,  vendor  maturity  is  a  limiting  factor  for  technologies  that  can  be  considered  for  adoption. 
The  U.S.  General  Services  Administration’s  IT  Schedule  70^  requires  that  cybersecurity  companies  have  at 
least  two  years  of  past  performance  for  consideration  as  a  vendor.  This  is  a  high  bar  that  limits  the  ability 
of  young  cybersecurity  companies  to  market  the  most  up-to-date  technologies  and  products  to  the 
government  to  mitigate  current  threats. 

2.4  THE  PURCHASE  PROCESS 

Procuring  expensive  technologies  that  involve  vendor  contracts  introduces  layers  of  paperwork  and 
intentional  technology  transition  delays;  the  process  can  be  rather  inflexible  to  address  changing  needs 
during  the  procurement  process.  According  to  Moore  et  ah,  “structural  issues  within  the  bureaucracy  often 
inhibit  adequate  and  timely  prioritization”  [2].  The  US  Government,  in  particular,  requires  a  full 
technology  acquisitions  approval  cycle  of  3  years  for  any  purchases  over  $3000.  Purchases  of  $500,000 
and  up  require  Congressional  approval.  By  the  time  these  technologies  have  been  purchased  and  deployed, 
the  entire  cyber  threat  domain  has  changed  and  the  solution  that  was  time  consuming  and  costly  to  procure 
is  now  effectively  outdated. 


®http://www.gsa.gov/schedule70 
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3.  DITEC  AS  A  PROOF-OF-CONCEPT 

The  Department  of  the  Navy  funded  the  DoD-DITEC  projeet  (and  following  efforts)  to  ereate  a  tool  to 
assist  aequisitions  personnel  in  purehasing  already  eertified  and  aeeredited  eyberseeurity  teehnology 
appropriate  to  their  systems  [3] .  Engineers  and  IT  personnel  often  have  biased  opinions,  while  managers 
who  are  non-seeurity  experts  are  likely  to  understand  their  seeurity  needs  as  explained  by  a  vendor  sales 
representative.  DITEC  standardizes  the  eyberseeurity  aequisition  proeess  in  part  by  instituting  guidelines 
and  frameworks  (e.g.,  NIST)  Cyberseeurity  Eramework,  the  DoD  8500  Series  Information  Assuranee 
Controls’)  to  establish  the  types  of  proeedures,  eontrols,  threats,  and  features  that  provide  the  test  eases  for 
whieh  eyberseeurity  teehnologies  would  be  evaluated.  A  market  survey  was  eondueted  to  learn  what 
teehnologies  were  available  for  aequisition  and  to  elassify  them  into  a  three-tiered  eategorization  based  on 
their  eapabilities.  Teehnology  evaluation  metries  and  a  seoring  algorithm  were  developed  by  ereating  a 
taxonomy  whieh  matehed  teehnologies  and  test  eases,  allowing  users  to  evaluate  and  make  high-level 
eomparisons  of  multiple  teehnologies  against  one  another. 

DITEC  eonsists  of  three  eomponents  [3] : 

1.  Proeess  -  Evaluates  a  speeifie  eyberseeurity  teehnology  to  determine  how  well  it  meets 
DoD/Navy  needs. 

2.  Metries  -  Measures  how  well  eaeh  teehnology  meets  the  speeified  needs  aeross  125  different  test 
eases. 

3.  Eramework  -  Provides  the  format  neeessary  to  eompare  and  eontrast  multiple  teehnologies  of  a 
speeifie  eyberseeurity  area. 

Teehnologies  were  rated  by  metries  on  three  levels  of  granularity.  The  highest  level  of  granularity  is  the 
“Capability”  level.  There  are  10  of  these  Capabilities,  whieh  eorrespond  to  very  broad  ability  eategories 
(e.g..  Protect,  Respond,  Operations,  Lifecycle  Management).  The  “Sub-Capability”  level  is  the  middle 
level  of  granularity,  narrowing  the  foeus  from  the  Capability  level  (e.g.,  Proteet — Cryptographic  Support, 
Eifeeyele  Management — Cost  of  Extended  Vendor  Support).  Einally,  the  “Sub-Capability  Elements”  level 
ineluded  very  speeifie  fesf  eases. 


’http://www.dtic.mil/whs/directives/corres/pdf/850001_2014.pdf 
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4.  DITEC+:  A  SCALABLE,  ENTERPRISE-READY  TECHNOLOGY 

EVALUATION  CAPABILITY 

DITEC  served  as  a  proof-of-eoneept  but  lacked  the  scalability  required  for  further  development  and 
adoption.  DITEC+  was  then  funded  to  resolve  issues  of  scalability  and  create  an  enterprise-ready  tool  to 
aid  in  the  acquisition  process.  Among  the  improvements  to  DITEC’s  proof-of-concept,  DITEC-i-  improved 
on  each  of  the  three  components: 

1.  Process — DITEC -i-  prescribes  additional/customizable  steps  for  focused  evaluations  pertaining  to 
specific  stakeholders  and  offers  those  steps  as  a  library  of  evaluation  guidelines. 

2.  Metrics — DITEC -i-  revised  and  improves  on  the  DITEC  metrics  module  to  enable  technologies  to 
receive  a  “score”  based  on  their  evaluation  performance  against  the  metrics  and  provides  the 
ability  to  apply  “weights”  to  each  evaluation  per  specific  items  of  interest  identified  during  the 
process.  DITEC-i-  Metrics  provide  support  for  prioritizing  results  based  on  a  variety  of  different 
aspects. 

3.  Eramework — DITEC -i-  leverages  the  existing  DITEC  Eramework  but  ensures  that  it  is  ready  for 
enterprise-wide  use,  supporting  multiple  users  and  evaluations  by  adding  robustness  to  the 
database  and  evaluation  algorithms. 

These  and  other  improvements  enable  a  DoD/Navy-centric,  cost-effective  and  streamlined  evaluation  of 
various  cybersecurity  technologies  that  is  defined  by  a  process  that  is  standardized,  flexible,  repeatable, 
scalable,  and  contains  granular  metrics  (developed  in-house  with  subject  matter  expert  support). 
Additionally,  DITEC-t: 

•  Supports  multiple  and  concurrent  users  and  technology  evaluations, 

•  Provides  the  ability  to  compare  various  cybersecurity  technology  evaluations, 

•  Integrated  CAUEDRON  [5],  a  network  vulnerability  mapping  tool, 

•  Developed  new  metrics  for  measuring  differences  across  evaluations  and  technologies  while 
estimating  the  level  of  cybersecurity  provided, 

•  Developed  a  new  ranking/prioritization  mechanism  of  evaluated  technologies  based  on  user 
preferences  [6],  and 

•  Developed  and  integrated  a  recommender  system  to  assist  security  non-experts  in  deciding  which 
technologies  best  suit  their  situational  needs  [7] . 

4.1  PRIORITIZATION  OF  EVALUATIONS 

Recognizing  that  personnel  at  different  levels  of  an  organization  would  have  competing  priorities,  the 
User  Priority  Designation  (UPD)  [6]  was  developed  to  view  evaluations  in  light  of  contextual  priorities. 

Eor  example,  an  agency’s  comptroller  may  place  a  heavier  priority  on  the  lifetime  cost  of  a  product,  where 
a  network  administrator  would  be  more  concerned  with  the  ability  to  install  a  vendor  update  with  minimal 
system  downtime. 

The  UPD  tool  uses  a  scalable  weighting  scheme  to  give  certain  capabilities  (Capabilities, 
Sub-Capabilities,  or  Sub-Capability  Elements)  greater  priority  over  others.  This  flexibility  allows  for 
cybersecurity  acquisition  decisions  to  be  made  with  input  from  multiple  levels — from  system 
administrators  who  actively  oversee  network  operations,  to  the  management  who  are  ultimately  responsible 
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for  the  overall  health  of  an  organization.  For  instance,  a  system  administrator  may  prioritize  certain 
security  features  as  well  as  ease  of  product  installation  and  use,  while  the  chief  financial  officer  is  primarily 
concerned  wifh  fhe  long-ferm  cosf  of  purchasing  and  mainfaining  fhaf  producf.  The  UPD  framework 
enables  all  sfakeholders  fo  accurafely  voice  fheir  priorities  and  concerns  fo  reach  opfimal  acquisition 
decisions. 

The  UPD  works  as  follows,  fhe  user  chooses  a  priorify  level  for  each  capabilify  and  a  weighf  is  assigned 
fo  each  capabilify  based  on  fhe  assigned  priorify.  The  priorify  levels  rank  from  0  fo  5  as  follows: 

•  UPD  Rank  5  (Top  Priorify) 

•  UPD  Rank  4  (High  Priorify) 

•  UPD  Rank  3  (Moderafe  Priorify) 

•  UPD  Rank  2  (Low  Priorify) 

•  UPD  Rank  1  (Minimal  Priorify) 

•  UPD  Rank  0  (No  Priorify). 

All  N  capabilities  musf  be  assigned  fo  a  UPD  Level,  and  multiple  capabilifies  may  be  assigned  fo  fhe 
same  level.  When  mulfiple  capabilifies  are  assigned  fo  fhe  same  UPD  Rank,  a  Collision  occurs.  Any 
capabilifies  given  UPD  level  0  are  denofed  as  Exclusions  and  will  nol  be  weighfed,  fhus  fhere  are 
M  =  N  —  Exclusions  positively  weighfed  capabilifies.  There  are  fhen  n  =  M  —  Collisions  weighfs  fo 
be  assigned  by  fhe  UPD.  Finally,  fhere  are  consfanfs  mo  =  0,  mi, ...,  nin  which  represenf  fhe  number  of 
capabilifies  fo  be  weighfed  af  each  UPD  Level. 

Take  fhe  initial  remaining  weighf  Iq  =  1  and  parfifion  if  info  Jq  =  Io/{mo  +  mi)  parfs.  The  firsf  weighf 
wi  =  Jo  +  {Jo  ■  {M  —  mi)/M).  From  fhere  we  can  recursively  define 

li+l  —  li  UT-i-i-l  •  lUj+l, 

Ji  =  Ii/{mo  H - h  mj+i), 

Wi+i  =  Ji  +  {Ji  ■  {M  -  mi+i)/M). 


Figure  5  in  Secfion  5.3  shows  a  graphical  inferprefafion  of  user  preferences  versus  evaluation  “raw  scores” 
fhrough  fhe  use  of  fhe  UPD  fool. 

Using  DITEC+’s  UPD  framework,  fechnology  evaluafions  can  be  viewed  by  cybersecurify  professionals 
and  by  managemenf,  allowing  all  sfakeholders  wifhin  fhe  agency  fo  projecf  how  various  fechnologies  and 
producfs  will  meef  fheir  needs  according  fo  differing  criferia.  If  is  offen  fhe  case  fhaf  cybersecurify 
non-experfs  will  have  a  role  in  fhe  acquisifion  decision  making  process  and  so  we  invenfed  fhe  Technology 
Mafching  Tool  (TMT),  a  recommender  system  which  helps  fo  mafch  users  fo  fhe  fechnology  (or  suife  of 
fechnologies)  which  besf  mafch  fheir  needs  [7,  13].  The  TMT  is  discussed  in  greater  defail  in  Secfion  5.3.1, 
in  fhe  confexf  of  cybersecurify  for  operational  fechnology  environments. 
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5.  C-SEC:  SUPPORTING  CYBERSECURITY  IN  INDUSTRIAL  CONTROL 
SYSTEMS  AND  CRITICAL  INFRASTRUCTURE 

Critical  infrastructure  (e.g.,  power  grids,  water  treatment  plants)  have  had  a  multitude  of  cyber-attack 
vulnerabilities  discovered  as  they  are  becoming  increasingly  interconnected  [8].  Supervisory  Control  and 
Data  Acquisition  (SCADA)  systems  [9],  used  in  many  automated  processes  including  power  generation 
and  distribution,  are  of  particular  interest  to  cybersecurity  researchers.  These  systems  are  notoriously 
fragile  (e.g.,  timing  sensitivities)  and  many  cybersecurity  products  available  on  the  market  for  Information 
Technology  (IT)  systems  are  inappropriate  for  SCADA  systems  [10].  The  Cyber-SCADA  Evaluation 
Capability  (C-SEC)  is  an  instantiation  of  DITEC-i-,  designed  specifically  to  test  and  evaluate  the  suitability 
of  cybersecurity  technologies  for  SCADA  environments  [11].  Security  has  not  traditionally  been  a  concern 
for  SCADA  systems  because  different  manufacturers  employed  diverse  protocols,  however  as  protocols 
have  become  standardized  this  is  no  longer  the  case  [12]. 

C-SEC  supports  cybersecurity  decision  making  across  new  technologies  by  enabling  a  streamlined, 
flexible,  and  repeatable  evaluation  process  against  DoD-specific  needs  and  requirements.  Traditional 
security  evaluations  are  expensive,  requiring  time  and  resources  that  smaller-scale  projects  cannot  afford. 
Moreover,  these  evaluations  tend  to  be  non-repeatable  and  ultimately  lack  usability  and  applicability 
beyond  just  that  one  instance,  thus  jeopardizing  their  long-term  ROI  [3].  As  an  instantiation  of  DITEC-i-, 
C-SEC  has  three  main  components: 

1 .  A  software  evaluation  tool 

2.  A  laboratory  environment 

3.  An  online,  collaborative  environment. 

The  software  evaluation  tool  walks  non-SCADA  security  experts  through  a  quick,  high-level  evaluation 
process  for  determining  the  highlights  of  specific  technologies  of  interest.  The  laboratory  environment 
integrates  the  technology  of  interest  into  a  prescribed  configuration,  which  then  provides  a  more  detailed 
evaluation.  Easily,  the  online  collaborative  environment  serves  as  a  repository  of  past  evaluations  to 
facilitate  reuse  of  results. 

5.1  The  C-SEC  Software  Evaluation  Tool 

The  C-SEC  Evaluation  Tool  is  composed  of  DITEC’s  three  main  parts:  a  process,  metrics,  and  a 
framework  (Eigure  1)  [3,  11].  This  approach  provides  not  only  the  process  necessary  to  determine  if  a 
certain  technology  meets  DoD/Navy  needs,  but  also  provides  the  metrics  to  measure  how  well  those  needs 
are  met  and  a  framework  to  enable  the  comparison  of  multiple  technologies  of  interest. 
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Figure  1 .  The  C-SEC  evaluation  framework. 


5.1.1  The  C-SEC  Software  Evaluation  Process 

A  software  evaluation  process  is  the  first  major  step  in  the  C-SEC  approach  to  cybersecurity  decision 
support.  It’s  purpose  is  to  evaluate  a  specific  cybersecurity  technology  and  determine  if  it  meets  (or  not) 
DoD/Navy  needs.  This  evaluation  process  involves  the  following  five  steps: 

1 .  Market  survey  -  High-level  view  of  current  offerings  for  the  particular  cybersecurity  technology  area 
of  interest  as  well  as  a  defined  set  of  tests  to  determine  the  compatibility  of  those  offerings  with 
identified  needs. 

2.  Analysis  and  drill-down  -  Results  from  the  market  survey  are  analyzed  and  the  top  percentage 
(varies  based  on  each  study,  interest,  and  needs)  of  technologies  will  move  down  to  the  next  step. 

3.  Focused  testing  -  This  step  takes  the  technologies  identified  during  the  analysis,  and  drill-down  to 
test  them  more  rigorously  by  means  of  test  cases  that  go  beyond  the  high-level  used  during  step  1 . 
Often  times  a  simulated  test  environment  is  built  to  test  the  functionality  of  the  test  subjects.  In 
addition,  documentation  reviews  are  also  used  to  determine  if  needs  are  met. 

4.  Focused  analysis  -  This  step  evaluates  the  results  for  each  technology  tested  in  Step  3  and  apply 
metrics  to  determine  how  well  each  need  was  met. 

5.  Gaps  and  recommendations  -  The  analyzed  results  are  used  to  determine  current  technology  gaps 
and  suggest  recommendations  for  future  research. 

5.1.2  C-SEC  Metrics 

Metrics,  and  specifically  those  which  are  related  to  non-functional  aspects,  pose  a  difficult  problem  for 
cybersecurity  analysis  and  decision  makers.Traditional  approaches  to  measuring  are  not  well  suited  for 


aspects  like  security  and  usability  [3].  Consequently,  researchers,  industry  practitioners,  and  the 
government  lack  the  appropriate  tools  to  baseline  and  track  specific  characteristics  of  current  technologies. 
C-SEC  provides  metrics  support  that  is  applicable  for  security  and  usability  characteristics,  and  is  relevant 
to  academia,  industry,  and  government  sectors.  Specifically,  C-SEC  provides  metrics  across  three  areas  to 
make  its  results  repeatable: 

1 .  Metrics  Discovery  and  Application  -  To  develop  DoD-specific  security  metrics  and  apply  them  to 
C-SEC  Process  results. 

2.  Metrics  Manipulation  -  To  enable  manipulation  and  results  integrity. 

3.  Metrics  Visualization  -  To  enables  metrics  traceability  and  decision  making  support. 

5.1. 2.1  Metrics  Discovery  and  Appiication 

Metrics  must  first  be  developed  and  their  application  to  C-SEC  results  determined.  To  provide  relevant 
metrics  to  a  variety  of  cybersecurity  technologies  we  have  selected  ten  different  metrics  areas,  referred  to 
as  Capabilities  (Table  1).  These  Capabilities  represent  the  highest  level  of  granularity  and  cover  aspects 
across  two  main  areas.  Computer  Network  Defense  (CND)  concepts  as  well  as  Product-Eevel.  The 
CND-level  metrics  refer  to  the  basic  aspects  related  to  security — that  is,  how  well  does  a  technology 
support  the  protection,  monitoring  and  detection,  analysis,  planning,  and  response  to  threats  or  attacks.  The 
Product-Eevel  metrics  refer  to  aspects  more  commonly  associated  with  “day-to-day”  operations  of  a 
technology.  Product-Eevel  metrics  look  at  aspects  that  range  from  the  cost  and  difficulty  of  deploying  a 
specific  technology  to  the  complexity  of  maintaining  that  technology  once  it  is  deployed.  Each  type  of 
metric  applied  to  the  results  obtained  from  the  application  of  the  C-SEC  Process  is  assigned  a  numerical 
value  that  reflects  how  well  the  specific  technology  under  evaluation  meets  the  objectives  defined  for  that 
metric. 


Table  1 .  C-SEC  capabilities,  divided  between  CND  and  product  level. 


Capabilities 

CND  Level 

Product  Level 

Protect 

Product  deployment 

Monitor/detect 

System  capabilities* 

Analyze 

Performance* 

Plan 

Operations 

Respond 

Lifecycle  management 

5.1. 2. 2  Metrics  Manipulation 

C-SEC  supports  the  manipulation  of  these  metrics  to  better  understand  the  technology  under  various 
shades  of  light.  C-SEC  employs  a  granular  approach  to  metrics  manipulation;  this  enables  flexibility  as 
well  as  reusability  of  results.  Eor  example,  suppose  that  Agency  1  just  completed  an  evaluation  of 
Technology  X  with  an  emphasis  on  the  cost,  but  now  Agency  2  also  wants  to  evaluate  the  same  Technology 
X  with  a  different  emphasis  on  protection  capabilities.  Agency  2  could  reuse  the  same  C-SEC  results  that 
Agency  1  produced,  and  manipulate  the  C-SEC  Metrics  to  put  more  weight  into  the  protection  aspects  of 
the  results  (and  less  on  the  cost  aspects)  to  obtain  a  different  measurement  of  technology  X’s  ability  to  meet 
those  needs. 
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C-SEC  Metrics  prescribe  two  new  levels  in  addition  to  the  Capability-level  described  in  Section  5.1.2, 
which  further  break  down  each  Capability  into  Sub-Capabilities,  and  those  into  Sub-Capability  Elements. 
This  granular  approach  prescribes  a  few  rules: 

•  Every  Capability  is  composed  of  one  or  more  Sub-Capabilities 

•  Every  Sub-Capability  is  composed  of  one  or  more  Sub-Capability  Elements 

•  Sub-Capability  Elements  can  be  duplicated  across  other  Sub-Capabilities. 

To  illustrate  this,  Eigure  2  shows  how  a  Capability,  like  Protection,  is  composed  of  two  Sub-Capabilities: 
Vulnerability  Protection  and  Eisting  (which  refer  to  two  possible  ways  to  achieve  protection).  These  are 
further  broken  into  Sub-Capability  Elements,  such  as  Vulnerability  Scanning  and  Vulnerability  Reporting 
(which  refer  to  two  possible  ways  to  achieve  Vulnerability  Protection). 


Capability 


j  WEIGHT  j 


1.  Protection 


SCORE 


Sub- 

Capability 


rzi 


1.1  Vulnerab. 
Prevention 


Sub- 

Capability 

Element 


1.2  Listing 

J  s  1 

1.1.1  Vulnerab. 
Scanning 

!  w]  LIJ 

1.1.2  Vulnerab. 
Reporting 

s 


1.2.1 

Blacklisting 


tjzkli^t 


Figure  2.  C-SEC  metric  granuiarity. 


C-SEC  computes  an  aggregated  score  from  various  levels  of  granularity  (Capability  — Sub-Capability 
— )•  Sub-Capability  Element)  as  well  as  provides  “weights”  at  each  element  to  facilitate  the  flexibility  and 
reuse  of  the  C-SEC  Metrics.  This  granular  system  is  what  would  enable  Agency  2,  in  our  earlier  example, 
to  take  the  C-SEC  Process  results  from  Agency  1  and  apply  different  weights  to  their  scores  to  emphasize 
different  aspects  of  interest. 

5. 1.2. 3  Metrics  Visualization 

The  final  aspect  supported  by  C-SEC  Metrics  is  visualization  (Eigure  3).  C-SEC  metrics  provide  a 
visualization  for  the  manipulation  of  the  various  scores  and  weights  applied  to  the  C-SEC  process  results  so 
that  users  can  see  in  real-time  the  effect  that  changes  have  on  the  original  results.  The  metrics  visualization 
component  is  mainly  driven  by  C-SEC ’s  graphical  user  interface  (GUI)  and  changes  made  to  the  original 
results  are  stored  in  a  database.  Einally,  the  visualization  of  C-SEC  metrics  also  supports  decision  making 
by  employing  Bayesian-network  models  to  provide  probabilities  as  well  as  ROI  information. 
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Figure  3.  C-SEC  evaluation  results. 


Weight 


5.1.3  C-SEC  Framework 

The  C-SEC  Framework  provides  the  format  necessary  to  compare  and  contrast  multiple  technologies  of 
a  specific  cybersecurity  area.  Furthermore,  the  framework  also  supports  a  repository  of  past  and  present 
evaluation  results  to  facilitate  reuse.  The  C-SEC  Framework  serves  as  the  key  component  of  the  online 
collaborative  environment  (to  be  discussed  in  Section  5.3),  through  which  various  users  can  share  results 
and  reuse  information.  While  C-SEC  applications  are  individually  installed  by  users  (clients),  the 
framework  serves  as  the  hub  (server)  that  connects  them  together. 

5.2  The  C-SEC  Laboratory  Environment 

To  give  more  meaningful  evaluations,  we  developed  the  C-SEC  Eaboratory  Environment.  The 
C-SECEaboratory  Environment  consists  of  several  SCADA  demonstration  kits  from  various  vendors, 
which  are  easily  reconfigurable  to  simulate  different  environments  and  are  available  from  various  vendors. 
Cyber  security  professionals  can  use  this  setup  to  investigate,  the  Technology  Under  Evaluation  (TUE)  for 
a  much  more  detailed  evaluation  beyond  just  the  C-SEC  software  tool.  Eaboratory  assets  include: 

•  SCADA  and  ICS  components  including,  but  not  limited  to  programmable  logic  control  units  (PECs), 
networking  equipment,  valves,  actuators,  motors,  and  various  other  components,  which  create  a 
realistic  industrial  environment,  these  components  are  representative  of  what  is  offered  by  the  major 
SCADA  and  ICS  suppliers 

•  A  DoD-mandated  vulnerability  scanner 

•  A  vulnerability  visualization  tool  called  Combinatorial  Analysis  Utilizing  Logical  Dependencies 
Residing  on  Networks  (CAULDRON)  [6] 

•  A  suite  of  internally  developed,  custom  security  test  scripts  to  exercise  the  equipment  beyond  normal 
operation  parameters. 
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CAULDRON,  developed  by  George  Mason  University,  is  a  network  vulnerability  visualization  tool  that 
takes  vulnerability  scan  results  in  .  xml  format,  parses  them,  and  outputs  a  weighted  network  diagram  [5]. 
The  nodes  represent  the  Internet  Protocol  (IP)  addresses  within  the  network  and  the  edges  show  potential 
for  information  exchange.  Each  edge  has  an  associated  weight  that  represents  the  number  of  vulnerabilities 
between  connected  nodes  and  shows  how  they  propagate  throughout  the  network.  This  tool  was  designed 
to  enable  network  modeling;  a  user  could  visualize  their  existing  network,  then  model  how  a  security 
product  would  change  the  state  of  the  network  based  on  placement.  A  set  of  scripts  was  developed  to 
automate  a  number  of  well-known  approaches  to  network  penetration.  These  scripts  are  deployed  on  the 
SCADA  network  to  test  the  effectiveness  of  security  products,  see  Figure  4. 
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Figure  4.  The  C-SEC  Laboratory  Evaluation  Process 


The  C-SEC  process  for  testing  the  effectiveness  of  security  technologies  is  as  follows  (Figure  4): 

1.  A  baseline  vulnerability  scan  of  SCADA  equipment  is  performed 

2.  The  TUE  is  installed  and  configured 

3.  Equipment  is  run  for  several  weeks  to  generate  data 

4.  SCADA  equipment  is  re-scanned 

5.  Scan  results  are  visualized  using  CAUEDRON 

6.  A  comparison  of  the  baseline  vulnerability  scan  and  re-scan  determines  the  effectiveness  of  the 
TUE 

7.  Internally  developed  security  test  scripts  are  performed  to  validate  scan  results. 
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5.3  The  C-SEC  Online  Collaborative  Environment 


The  C-SEC  Online  Collaborative  environment  consists  of  a  web  application  providing  users  with  an 
interface  to  create,  search,  and  reuse  standardized  evaluations  of  security  technologies  that  are  specifically 
marketed  for  SCADA  networks.  This  online  collaborative  environment  was  created  to  standardize  what  is 
currently  a  disparate  process  for  evaluating  security  technologies  and  apply  a  set  of  weights  that  is 
representative  of  user  needs.  Users  have  the  option  to  perform  new  evaluations  or  choose  previously 
completed  evaluations.  Allowing  users  to  choose  existing  evaluations  enables  them  to  make  informed 
decisions  about  which  technology  (or  suite  of  technologies)  to  implement  on  their  networks.  The  online 
environment  is  designed  for  easy  deployment  to  both  enterprise  and  tactical  networks.  To  achieve  this  goal, 
C-SEC  deploys  virtual  machines  (VM)  that  are  lightweight  and  can  to  be  hosted  in  almost  any  environment. 

One  of  the  most  valuable  sources  of  information  for  choosing  security  controls  is  peer  feedback,  and  as 
such  an  overarching  goal  for  C-SEC  is  the  development  of  a  repository  of  evaluations  for  reuse  [10].  When 
a  user  wants  to  reuse  an  existing  evaluation,  they  have  two  options.  They  can  take  another  user’s  evaluation 
at  face  value;  they  are  satisfied  with  the  answers  provided  by  the  other  user  and  accept  the  score. 
Alternatively,  they  can  reuse  existing  evaluations  while  overlaying  a  set  of  weights  based  on  individual 
needs  using  the  built-in  wizard  to  establish  their  preferences  [6].  Weights  based  on  an  individual’s 
prioritization  are  overlaid  onto  the  existing  evaluation  data.  The  user  also  sees  a  visualization  consisting  of 
a  bar  graph  of  existing  scores  over  each  of  the  Capabilities  and  an  overlay  “wave”  of  their  weighted 
preferences.  This  allows  users  to  directly  compare  their  priorities  to  what  the  technology  provides 
according  to  the  existing  evaluation,  shown  in  Eigure  5. 


Figure  5.  The  C-SEC  “Wave”  Overlay  comparing  user  priorities  with  existing  evaluations. 
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5.3.1  The  Technology  Matching  Tool:  A  Recom mender  System  for  Security  Non-Experts 

A  Technology  Matching  Tool  (TMT)  was  implemented  in  C-SEC’s  Online  Collaborative  Environment 
as  a  recommender  system  to  assist  security  non-experts  make  acquisition  decisions  when  determining  the 
cybersecurity  technologies  that  best  suit  their  needs  [13].  The  TMT  makes  use  of  the  UPD  introduced  in 
Section  4.1.  The  Sub-Capabilities  described  in  Section  5.1.2  offer  enough  specificity  to  direct  a  user  to 
specific  fechnology.  The  TMT  uses  fhis  feafure,  along  wifh  fhe  Sub-Capabilify  Elemenf  use  cases,  fo 
recommend  cybersecurify  fechnologies  fhaf  are  appropriafe  fo  fhe  nefwork  in  quesfion.  There  are  fwo 
phases  fo  fhe  TMT: 

1.  Sub-Capabilify  Priorifizafion  -  differenf  Sub-Capabilifies  will  have  differing  levels  of  imporfance 
wifh  changing  confexf.  The  TMT  uses  fhe  UPD  framework  (wifh  a  less  developed  weighfing 
scheme)  fo  assign  weighfs  according  fo  fhe  Sub-Capabilify’s  priority  in  a  given  confexf. 

2.  Use  Case  Specificafion  -  use  cases,  in  fhe  form  of  Sub-Capabilify  Elemenfs,  are  defermined. 

The  user  assigns  a  priorify  level  according  fo  fhe  UPD  framework  fo  fhe  (currenfly  44)  Sub-Capabilifies 
fhaf  C-SEC  uses.  These  Sub-Capabilifies  are  weighfed  according  fo  fheir  assigned  priorify  (e.g.,  a 
Sub-Capabilify  wifh  a  UPD  rank  of  5  will  have  a  weighl  of  1.00,  while  a  Sub-Capabilify  wifh  a  rank  of  3 
will  have  a  weighl  of  0.55).  The  use  case  confexf  requiremenls  are  specified  by  fhe  user  completing  a  series 
of  yes/no  quesfions  based  on  fhe  (currenfly  109)  Sub-Capabilify  Elemenfs. 

Each  cybersecurify  fechnology  evaluafed  for  C-SEC  receives  a  Sub-Capabilify  Elemenf  profile  which  is 
“vecforized”.  This  gives  a  binary  “Producf  Vector”  where,  in  fhe  even!  fhaf  fhe  technology  performs  a 
Sub-Capabilify  Elemenf  use  case  fhe  corresponding  veclor  elemenf  is  a  1,  and  a  0  olherwise. 
Correspondingly,  a  binary  “User  Requiremenf  Vector”  is  derived  from  fhe  yes/no  quesfionnaire  fhaf  was 
used  fo  specify  confexfual  use  case  requiremenls.  The  veclor  elemenfs  are  weighfed  according  to  fhe 
priorify  weighfing  of  fheir  corresponding  Sub-Capabilify  Elemenfs,  fhaf  receive  a  weigh!  according  to  fheir 
Sub-Capabilify.  A  weighted  Euclidean  Melric  is  Ihen  used  fo  malch  fhe  user  wifh  fhe  fechnologies  fhaf  besl 
suils  fheir  defermined  needs  [14]. 

The  TMT’s  basic  funclionalily  can  be  modified  fo  improve  fhe  user  experience.  Eor  example,  a 
Sub-Capabilify  receiving  a  UPD  rank  of  0,  may  have  ils  associaled  use  cases  disregarded  in  fhe  Use  Case 
Specificafion  phase.  Eigure  6  displays  fhe  TMT’s  resulls  for  single  fechnologies,  bul  fhe  TMT  can 
recommend  suites  of  cybersecurify  fechnologies  fhaf  will  work  in  concerl  fo  meel  fhe  nelwork’s  confexfual 
requiremenls  (Eigure  7). 
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Single  Technology  Best  Fit 


Network  Based  IPS  92/100 

Security  Information  and  Event  91/100 

Manaoement  /SIEM)  system 

Activitv/Behavior  Scanners  90/100 

Signature  Scanners  89/100 

Heuristic  Scanners  89/100 

Host  Based  IPS  88/100 

AoDlication  Based  IPS  85/100 

Fault  Management  84/100 

Security  Management  83/100 

Anti-Malware  and  Program  Control  83/100 


Figure  6.  The  C-SEC  TMT,  offering  singie  technoiogy  recommendations  based  on  Sub-Capabiiity 
Prioritization  and  Use  Case  Specification. 


Technology  Suite  Best  Fit 


Inline  Media  Encryptor 
Network  Based  IPS 


Inline  Media  Encrvotor 
Security  Information  and  Event 
Management  fSIEM)  system 


Aoplication  Security  Management 
Security  Information  and  Event 
Management  (SIEM1  system 


Inline  Media  Encryptor 
Host  Based  IPS 


VPN 

Security  Information  and  Event 
Management  (SIEM1  system 


Network  Based  IPS 
HAIPEs 


Inline  Media  Encrvotor 
Signature  Scanners 


Network  Based  IPS 


Storage  Device 


VPN 

Application  Based  IPS 


Network  Based  IPS 
VPN 


97/100 


96/100 


95/100 


95/100 


95/100 


95/100 


95/100 


95/100 


Figure  7.  The  C-SEC  TMT,  offering  technoiogy  suite  recommendations  based  on  Sub-Capabiiity 
Prioritization  and  Use  Case  Specification. 
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6.  FUTURE  WORK 


DITEC  was  a  proof-of-concept  that  provided  an  early  exploration  at  eyberseeurity  metries,  objeetive 
evaluations,  and  re-usability  of  sueh  evaluations.  While  still  experimental  systems,  DITEC-i-  and  C-SEC 
became  much  more  mature  projects  which  enabled  actual  evaluations  of  eyberseeurity  products  and 
developed  tools  for  comparing  their  results.  Eor  the  DITEC  process  to  have  a  broader  impact,  it  must  be 
transitioned  from  an  experimental  program  to  a  “Program  of  Record”  to  guide  eyberseeurity  acquisition 
across  agencies.  This  transition  would  ultimately  enable  not  just  a  wider  adoption,  but  lead  to  a 
standardization  of  eyberseeurity  metrics  and  evaluations  across  the  government  sector.  Earger  private 
institutions  such  as  banks  or  chain  retailers  may  also  benefit  from  the  adoption  of  DITEC ’s  process, 
metrics,  and  recommendations  for  successful  deployments. 

Continuing  work  is  underway  to  make  the  DITEC  family  suitable  for  larger  adoption,  including  the 
following: 

•  Eurther  development  of  metrics  and  test  cases  to  evaluate  “active”  eyberseeurity  and 
counterintelligence  technologies  (e.g.,  honeypots) 

•  Development  of  instances  of  DITEC-i-  for  environments  beyond  the  enterprise  and  SCADA 
environments  (e.g.,  loT  and  Eog  Computing  environments) 

•  Study  into  how  DITEC  process  results  can  be  turned  into  actionable  plans  that  could  enable  much 
more  automated  cyber  defenses 

•  Enhance  the  DITEC  and  C-SEC  concepts  with  ROI  as  it  relates  to  specific  areas  of  interest  (e.g., 
energy,  internet  of  things  (loT),  etc) 

•  Human  factors  evaluation  study  to  improve  the  user  experience  as  well  as  the  overall  usability  of 
the  processes  and  tools. 

The  UPD  and  TMT  show  promise  in  helping  determine  the  ROI  of  eyberseeurity  acquisition  decisions. 
A  challenge  of  understanding  ROI  for  eyberseeurity  is  that  it  is  difficult  to  quantify.  There  are  many 
reasons  for  this  optimism.  Eor  instance  the  impact  of  purchasing  and  integrating  a  technology  must  be 
weighed  against  hypothetical  circumstances  as  well  as  potentially  outdated  information  regarding  current 
states.  Many  decision  makers  see  ROI  for  eyberseeurity  as  either  impossible  to  calculate  or  simply  do  not 
factor  it  into  their  acquisition  decisions  [2].  With  some  modification,  the  TMT  could  be  used  to  illustrate 
ROI  of  eyberseeurity  technologies  against  monetized  impacts. 

We  are  working  on  “light’Vmobile  versions  of  our  tool  sets  to  enable  evaluations  to  happen  where  the 
systems  are,  rather  than  just  in  our  laboratory  setting.  We  expect  this  an  area  will  eventually  expose  our 
work  to  new  opportunities  and  requirements. 

Beyond  the  current  plans  for  enhancing  current  process  and  tools  such  as  UPD  and  TMT,  we  foresee  a 
need  to  expand  and  generalize  our  approach  to  become  applicable  to  other  important  areas  like  safety  and 
privacy.  The  ability  to  not  only  make  better-informed  decisions  that  matter  today,  but  also  enable  modeling 
of  future  effects  of  such  decisions  is  important  for  future  work.  One  of  the  challenges  we  currently  face  is 
the  general  aversion  to  improvements  that  seem  like  standards  (and/or  the  attempt  to  standardize 
something)  in  the  age  where  prototyping  (i.e.,  failing  early  and  often)  reigns.  We  understand  this  challenge 
and  seek  to  continue  the  development  of  our  processes  and  tool  sets  to  go  beyond  simply  pointing  at 
something  as  right  or  wrong,  but  enabling  users  to  make  actionable  decisions  that  are  reusable. 
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7.  CONCLUSION 


The  U.S.  Government  faees  diffieult  obstaeles  with  respeet  to  eyherseeurity  teehnology  aequisition. 
Some  of  these  problems  relate  to  the  rigidity  of  budgetary  eyeles,  whieh  plan  several  years  in  advanee  and 
require  signifieant  effort  to  alter.  Limited  funds  to  support  training  and  workforee  development  often  mean 
sending  a  single  employee  to  training  and  having  them  return  to  train  their  eolleagues  on  what  they  have 
just  been  taught.  Other  aequisition  polieies  make  purehasing  newly  developed  teehnologies,  that  eould 
effeetively  defend  against  emerging  threats,  virtually  impossible.  These  diffieulties,  partieularly  the 
long-term  budgeting  and  aequisition  eyeles,  make  the  government  an  attraetive  potential  eustomer  for 
eyherseeurity  teehnology  eompanies,  and  many  have  sales  teams  (often  eomposed  of  retired  government 
personnel)  speeializing  in  publie  seetor  eustomers.  This  situation  ean  lead  to  a  “fog  of  more”  situation 
where  eyherseeurity  aequisition  deeisions  are  made  with  an  avalanehe  of  inadequate  information. 

Even  seasoned  IT  professionals  may  have  ineomplete  knowledge  or  biased  opinions  that  hinder  the 
effeetiveness  of  eyherseeurity  aequisition.  DITEC  and  its  family  of  produets  provide  aequisition  deeision 
support  for  eyherseeurity  teehnologies  and  produets  that  have  reeeived  eertifieation  and  aeereditation. 
DITEC  and  its  follow-on  works,  DITEC -i-  and  C-SEC,  demonstrate  the  ability  to  mitigate  this  problem  by 
building  a  teehnology  evaluation  proeess  and  tool  that  is  standardized  and  repeatable.  DITEC-i-  and  its 
instantiations  offer  the  ability  to  store  and  reuse  produet  evaluations,  weighted  to  relieet  individual 
priorities.  Institutions  with  higher  personnel  turnover  will  suffer  a  loss  of  institutional  knowledge,  possibly 
leaving  aequisitions  deeisions  to  be  made  by  eyherseeurity  non-experts.  A  TMT  works  as  a  reeommender 
system  that  assists  seeurity  non-experts  to  the  teehnologies  that  will  best  meet  the  needs  of  their  networks. 
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